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This  article  summarizes  the  results  of  a  Defense  Science  Board  (DSB)  Task  Force  study  of 
Developmental  Test  and  Evaluation  (DT^E)  (Department  of  Defense,  May  2008),  which 
was  conducted  in  2007  and  early  2008.  The  purpose  of  the  study  was  to  investigate  the  causal 
factors  for  the  high  percentage  of  programs  entering  Initial  Operational  Test  and  Evaluation 
(lOT^E)  in  recent  years  which  have  not  been  evaluated  as  both  operationally  effective  and 
operationally  suitable.  The  following  is  a  summary  of  the  specific  issues  which  the  Task  Force 
was  asked  to  assess: 

•  Office  of  the  Secretary  of  Defense  (OSD)  organization,  roles,  and  responsibilities  for  Test  and 
Evaluation  (Tl^E)  oversight.  Recommend  changes  that  may  contribute  to  improved  DT^E 
oversight,  and  facilitate  integrated  T^E; 

•  Changes  required  to  establish  statutory  authority  for  OSD  DT^^E  oversight.  Recommend 
changes  to  Title  10  or  other  US.  statutes  that  may  improve  OSD  authority  in  DTitfE  oversight; 

•  Many  lOT&'E  failures  have  been  due  to  lack  of  operational  suitability.  Recommend 
improvements  in  DTitfE  process  to  discover  suitability  problems  earlier,  and  thus  improve 
likelihood  of  operational  suitability  in  lOTitfE. 
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In  recent  years,  there  has  been  a  dramatic 
increase  in  the  number  of  systems  not 
meeting  suitability  requirements  during  Ini¬ 
tial  Operational  Test  and  Evaluation 
(lOT&E).  Reliability,  availability  and  main¬ 
tainability  (RAM)  deficiencies  comprise  the  primary 
shortfall  areas.  Department  of  Defense  Initial  Opera¬ 
tional  Test  and  Evaluation  (DoD  lOTScE)  results 
from  2001  to  2006  are  summarized  in  Figures  1 
through  3.  These  charts  graphically  depict  the  high 
suitability  failure  rates  during  lOT&E  resulting  from 
RAM  deficiencies. 


Early  in  the  Defense  Science  Board  (DSB)  study,  it 
became  obvious  that  the  high  suitability  failure  rates 
were  the  result  of  systemic  changes  that  had  been  made 
to  the  acquisition  process;  and  that  changes  in 
developmental  test  and  evaluation  could  not  remedy 
poor  program  formulation.  Accordingly,  the  Task 
Force  study  was  expanded  to  address  the  broader 
programmatic  issues,  as  well  as  the  above  issues 
identified  in  the  Terms  of  Reference  (TOR). 

A  number  of  major  changes  in  the  last  15  years  have 
had  a  significant  impact  on  the  acquisition  process. 
First,  Congressional  direction  in  Fiscal  Year  (FY) 
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Program 

Service 

ACAT 

lOT&E  Result 

Reason 

FY2001 

F-15  TEWS 

USAF 

II 

Effective 

itet^uitable 

Reliability,  Maintainability,  Availability 

V-22  Osprey 

Navy 

ID 

Effective 

NBt  Suitable 

Reliability,  Availability,  Maintainability 
(RAMX  Human  Factors,  BIT 

Joint  Direct  Attack 

Munitions  (JDAM) 

USAF 

1C 

Effective  only  with 
legacy  fuses 

nm  SuifeUe 

Integration  with  delivery  platforms 

M2A3  Bradley  Fighting 
Vehioie 

Army 

ID 

Effective 

Suitable 

FY2002 

Joint  Primary  Aircraft 
Training  System  (JPATS) 

USAF 

1C 

Effective  with 
deficiencies 

N^SuHabie 

RAM,  Safety,  Human  Factors 

Cooperative  Engagement 
Capability  (CEC) 

Navy 

ID 

Effective 

Suitable 

Multiple  Rocket  Launcher 
System  (MLRS) 

Aniiy 

1C 

Effective 

Suitable 

MH«0S 

Navy 

1C 

Effective  ^^^^^^^HRAM,  excessive  administrative  and  logistic 

repair  time  impacted  RAM 

FY2003 

B-1B  Block  E  Mission 

Upgrade  Program 

USAF 

ID 

f : 

16% decrease  in  weapons  release  rate, 

reduction  In  accuracy  of  Mark  82  low  drag 
weapons,  14%  hit  rate  on  moving  targets 

Sea  wolf  Nuclear  Attack 

Submarine 

Navy 

ID 

Effective 

Suitable 

Several  requirement  thresholds  were  not 
met  but  overall  system  effective  andsultabh 

Figure  1.  DoD  iOT&E  results  FY  2001-2003 


1996,  1997,  1998,  and  1999  Defense  Authorization 
Acts  reduced  the  acquisition  workforce  (which  includes 
developmental  test  and  evaluation).  Several  changes 
resulted  from  the  implementation  of  Acquisition 


Reform  in  the  late  1990s.  The  use  of  existing 
commercial  specifications  and  standards  was  encour¬ 
aged,  unless  there  was  justification  for  the  use  of 
military  specifications.  Industry  was  encouraged  to  use 


Program 

Service 

ACAT 

lOT&E  Result 

Reason 

FY  2004 

Evolved  Sea  sparrow  Missile 

Navy 

II 

Effectiveness 

unresolved 

Suitable 

Testing  was  not  adequate  to 
determine  effectiveness. 

Strylcer 

Army 

ID 

Effective 

Suitable 

Advanced  SEAL  Delivery  System 
(ASDS) 

Navy 

ID 

Effective  with 
restrictions 

Effective  for  short  duration 
missions;  not  effective  for  all 
missions  and  profiles. 

Not  suitable  due  to  RAM. 

Taotloal  Tomahawk 

Navy 

1C 

EffeoUve 

Suitable 

Stryker  Mortar  Carrlar-B  (MC-B) 

Army 

ID 

Effective 

Not  SoteaMa 

RAM  and  safety  eoncems. 

FY2005 

CH-47F  Blook  1 

Army 

1C 

Effective 

HabSuHsMiK 

RAM;  communications  system  less 
sultaMe  than  CH-47D;  did  not 
meet  Information  Exchange 
Requirements  for  Block  I. 

F/A-22 

USAF 

ID 

Effective 

Mot 

RAM;  needed  more  maintenance 
resources  and  spare  parts;  BIT 

Joint  Stand-Off  Weapon-C 

Navy 

1C 

NKflCecHue 

Not  effective  against  moderately 
hardened  targets;  mission 
planning  time  was  excessive. 

Gulded-MLRS 

Army 

1C 

Effective 

Suitable 

High  Mobility  Attack  Rocket  System 
(HMARS) 

Army 

1C 

Effective 

Suitable 

V-22  Osprey 

Navy 

1D 

Effective 

Suitable 

EA-6B  (ICAP  III) 

Navy 

II 

Effective 

Suitable 

Figure  2.  DoD  lOT&E  results  FY  2004-2005 


216  ITEA  Journal 


The  ITEA  Journal  of  Test  and  Evaluation  jite-29-03-15.3d  21/8/08  12:28:05  216 


Defense  Science  Board  Task  Force  Developmental  Test  and  Evaluation  Study  Results 


Program 

Service 

ACAT 

lOTSE  Result 

Reason 

CY200* 

Common  Missile  VN^rning 
System  (CMWS) 


OepioyaWe  Joint  Commend  and 
Control  (DJC2) 


Integrated  Defensive  biectromc 
Countermeasures 


Surface  Electronic  Warfare 
Improvement  Program  (SEWIP) 


Small  Diameter  Bomb  (SOB) 
Increment  1 


Army 


Navy 


Navy 


Navy 


Effective  and  suitable  in  the  OIF/OEF  environment 
but  needs  further  testing  outside  of  the  OIF /OEF 
environment 


Operational  Test  Agency,  COTT,  reported  effective, 
notsutable.  BLRIPnotcontplete. 


lest  suspended  due  to  reliabilit/  problems. 


Block  1 A  Upgrade  does  not  make  the  AN/SLO-32 
EWS  opcratior^ally  effective  and  suitable  but  ctoes 
enhance  ability  to  protect  ships 


Effective  single  ship;  not  effective  in  formation  air  land 
/  air  drop,  not  eff^rve  in  non  permissive  threat 
environmert  Shortfalls  in  suitability  due  to 
maintainability  issues 


Limited  effectiveness  and  suitability  due  to  bomb  rack 
reliability  and  deficiencies  in  software  used  to  predict 
optimum  fuzing  solutions  Oct  2006  flight  operations 
suspended 


Figure  3.  DoD  lOT&E  results  for  2006 


commercial  practices.  Numerous  military  specifications 
and  standards  were  eliminated  in  some  Service 
acquisition  organizations.  The  requirement  for  a 
reliability  growth  program  during  development  was 
also  de-emphasized,  and  in  most  cases,  eliminated.  At 
the  same  time,  systems  became  more  complex,  and 
systems-of-systems  integration  became  more  common. 
Finally,  there  was  a  loss  of  a  large  number  of  the  most 
experienced  management  and  technical  personnel  in 
government  and  industry  without  an  adequate  replace¬ 
ment  pipeline.  The  loss  of  personnel  was  compounded 
in  many  cases  by  the  lack  of  up-to-date  standards  and 
handbooks,  which  had  been  allowed  to  atrophy,  or  in 
some  cases,  were  eliminated.  It  should  be  noted  that 
Acquisition  Reform  included  numerous  beneficial 
initiatives.  There  have  been  many  programs  involving 
application  of  poor  judgment  in  the  last  15  years  that 
can  be  attributed  to  acquisition/ test  workforce  inexpe¬ 
rience  and  funding  reductions.  It  is  probable  that  these 
problems  would  have  occurred  independently  of  most 
Acquisition  Reform  initiatives. 

All  Service  acquisition  and  test  organizations 
experienced  significant  personnel  cuts,  the  magnitude 
varying  from  organization  to  organization.  Over  time, 
in-house  DoD  offices  of  subject  matter  experts  (who 
specialized  in  multiple  areas,  such  as  promoting  the  use 
of  proven  reliability  development  methods)  were 
drastically  reduced,  and  in  some  cases,  disestablished. 


A  summary  of  reductions  in  developmental  test 
personnel  follows.  The  U.S.  Army  essentially  elimi¬ 
nated  their  military  developmental  testing  (DT) 
component  and  declared  the  conduct  of  DT  by  the 
government  to  be  discretionary  in  each  program.  The 
U.S.  Navy  reduced  their  DT  workforce  by  10  percent 
but  no  shift  of  “hands-on”  government  DT  to  industry 
DT  occurred.  The  trend  within  the  U.S.  Air  Force 
gave  DT  conduct  and  control  to  the  contractor.  Air 
Force  test  personnel  have  been  reduced  by  approxi¬ 
mately  15  percent  and  engineering  personnel  support¬ 
ing  program  offices  have  been  reduced  by  as  much  as 
60  percent  in  some  organizations.  The  reduction  of 
DT  personnel  in  the  Services  occurred  during  a  time 
when  programs  have  become  increasingly  complex 
(e.g.,  significant  increases  in  software  lines  of  code,  off- 
board  sensor  data  integration,  and  systems  of  systems 
testing). 

Principal  findings  and  recommendations 

RAM 

As  a  result  of  industry  recommendations  in  the  early 
1970s,  the  Services  began  a  concerted  effort  to 
implement  reliability  growth  testing  as  an  integral  part 
of  the  development  process.  This  implementation 
consisted  of  a  reliability  growth  process  wherein  a 
system  is  continually  tested  from  the  beginning  of 
development,  reliability  problems  are  uncovered,  and 
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corrective  actions  are  taken  as  soon  as  possible.  The 
Services  captured  this  practice  in  their  reliability 
regulations,  and  the  DoD  issued  a  new  military  standard 
on  reliability,  which  included  reliability  growth  and 
development  testing  as  a  best  practice  task.  The  goal  of 
this  process  from  1980  until  the  mid-1990s  was  to 
achieve  good  reliability  by  focusing  on  reliability 
fundamentals  during  design  and  manufacturing  rather 
than  merely  setting  numerical  requirements  and  testing 
for  compliance  towards  the  end  of  development. 

The  general  practice  of  reliability  growth  was  discon¬ 
tinued  in  the  mid  to  late  1990s,  concurrent  with  the 
implementation  of  Acquisition  Reform.  This  discon¬ 
tinuance  may  not  be  a  direct  result  of  Acquisition 
Reform,  but  may  be  related  instead  to  the  loss  of  key 
personnel  and  experience,  as  well  as  shortsighted  attempts 
to  save  acquisition  funds  at  the  expense  of  increased  life 
cycle  costs.  With  the  current  DoD  policy,  most 
development  contracts  do  not  include  a  robust  reliability 
growth  program.  The  lack  of  failure  prevention  during 
design,  and  the  resulting  low  initial  Mean  Time  Between 
Failure  (MTBF)  and  low  growth  potential  are  the  most 
significant  reasons  that  systems  are  faihng  to  meet  their 
operational  reliabiUty  requirements. 

Findings: 

Acquisition  personnel  reductions  combined  with 
acquisition  system  changes  in  the  last  15  years  had  a 
detrimental  impact  on  RAM  practices: 

-  With  some  exceptions,  the  practice  of  reli¬ 
ability  growth  methodologies  was  discontin¬ 
ued  during  system  design  and  development 

(SDD); 

-  Relevant  military  specifications,  standards, 
and  other  guidance  were  not  used; 

-  Suitability  criteria,  including  RAM,  were  de- 
emphasized. 

•  Improved  RAM  decreases  life  cycle  costs  and 
reduces  demand  on  the  logistics  system. 

•  The  deficiency  report  can  be  a  valuable  tool  for 
early  identification  of  RAM-related  suitabihty 
problems,  when  used  in  conjunction  with  an 
adequately  resourced  deficiency  correction  system. 

Recommendations: 

The  single  most  important  step  necessary  to  correct 
high  suitability  failure  rates  is  to  ensure  programs  are 
formulated  to  execute  a  viable  systems  engineering 
strategy  from  the  beginning,  including  a  robust  RAM 
program,  as  an  integral  part  of  design  and  develop¬ 
ment.  No  amount  of  testing  will  compensate  for 
deficiencies  in  RAM  program  formulation.  To  this 
end,  the  following  RAM-related  actions  are  required 
as  a  minimum: 


•  Identify  and  define  RAM  requirements  during 
the  Joint  Capabilities  Integration  Development 
System  (JCIDS)  process,  and  incorporate  them  in 
the  Request  for  Proposal  (RFP)  as  a  mandatory 
contractual  requirement. 

•  During  source  selection,  evaluate  the  bidders’ 
approaches  to  satisfying  RAM  requirements: 

-  Ensure  flow-down  of  RAM  requirements  to 
subcontractors, 

-  Require  development  of  leading  indicators  to 
ensure  RAM  requirements  are  met. 

•  Make  RAM,  to  include  a  robust  reliability 
growth  program,  a  mandatory  contractual  re¬ 
quirement  and  document  progress  as  part  of  every 
major  program  review. 

•  Ensure  that  a  credible  reliability  assessment  is 
conducted  during  the  various  stages  of  the 
technical  review  process  and  that  reliability 
criteria  are  achievable  in  an  operational  environ¬ 
ment. 

•  Strengthen  program  manager  accountability  for 
RAM-related  achievements. 

•  Develop  a  military  standard  for  RAM  develop¬ 
ment  and  testing  that  can  be  readily  referenced  in 
future  DoD  contracts. 

•  Ensure  an  adequate  cadre  of  experienced  RAM 
personnel  are  part  of  the  Service  acquisition  and 
engineering  office  staffs. 

Roles  and  responsibilities  of  government  test 
and  evaluation  organizations 

The  traditional  role  of  the  government  during  the 
DT  planning  phase  included  the  identification  of  the 
test  resource  requirements  and  government  test 
facilities,  the  development  of  the  test  strategy  and 
detailed  test  and  evaluation  plans,  as  well  as  the  actual 
conduct  of  T&E.  When  a  program  moved  from  the 
planning  phase  to  the  test  execution  phase,  the 
government  traditionally  participated  in  test  conduct 
and  analysis;  performing  an  evaluation  of  the  test 
results  for  the  program  office.  With  some  exceptions, 
this  is  no  longer  the  case.  Until  recently,  it  was 
recognized  that  there  should  be  some  level  of 
government  involvement  and  oversight  even  when 
the  contractor  has  the  primary  responsibility  regarding 
planning  and  execution  of  the  DT  program. 

Findings: 

The  changes  in  the  last  15  years,  when  aggregated, 
have  had  a  significant  negative  impact  on  DoD’s  ability 
to  successfully  execute  increasingly  complex  acquisition 
programs.  Major  contributors  include  massive  work¬ 
force  reductions  in  acquisition  and  test  personnel,  a 
lack  of  up-to-date  process  guidance  in  some  acquisition 
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organizations,  acquisition  process  changes,  as  well  as 
the  high  retirement  rate  of  the  most  experienced 
technical  and  managerial  personnel  in  government  and 
industry  without  an  adequate  replacement  pipeline. 

•  Major  personnel  reductions  have  strained  the 
pool  of  experienced  government  test  personnel. 

•  A  significant  amount  of  developmental  testing  is 
currently  performed  without  a  needed  degree  of 
government  involvement  or  oversight  and  in 
some  cases,  with  limited  government  access  to 
contractor  data. 

Recommendations: 

•  As  a  minimum,  government  test  organizations 
should  develop  and  retain  a  cadre  of  experienced 
T8cE  personnel  to  perform  the  following  functions: 

-  Participate  in  the  translation  of  operational 
requirements  into  contract  specifications,  and 
in  the  source  selection  process,  including  RFP 
preparation; 

-  Participate  in  Developmental  Test  and  Eval¬ 
uation  (DT&E)  planning  including  Test  and 
Evaluation  Master  Plan  (TEMP)  preparation 
and  approval; 

-  Participate  in  technical  review  processes; 

-  Participate  in  test  conduct,  data  analysis,  and 
evaluation  and  reporting;  with  emphasis  on 
analysis  and  reporting. 

•  Utilize  red  teams,  where  appropriate,  to  com¬ 
pensate  for  shortages  in  skilled,  experienced  T6cE 
domain  and  process  experts. 

•  Develop  programs  to  attract  and  retain  govern¬ 
ment  personnel  in  T&E  career  fields  so  that  the 
government  can  properly  perform  its  role  as  a 
contract  administrator  and  as  a  “smart  buyer”. 

Integrated  test  and  evaluation 

Integrated  testing  is  not  a  new  concept  within  the 
Department  of  Defense,  but  its  importance  in  recent 
years  has  been  highlighted,  due  in  part  to  the  growth  of 
asymmetric  threats  and  the  adoption  of  net-centric 
warfare.  The  December  2007  Office  of  the  Secretary  of 
Defense  (OSD)  Test  and  Evaluation  Policy  Revisions 
memorandum  reinforces  the  need  for  integrated 
testing.  Implementation  of  integrated  test  concepts 
has  been  allowed  to  evolve  on  an  ad  hoc  basis.  The  time 
has  come  to  pursue  more  consistency  in  integrated  test 
planning  and  execution. 

Collaboration  between  developmental  and  opera¬ 
tional  testers  to  build  a  robust  integrated  test  program 
will  increase  the  amount  of  operationally  relevant  data 
that  can  be  used  by  both  communities.  DT  and 
Operational  Test  (OT)  planning  is  separate  and  this 
inhibits  efforts  by  the  Services  to  streamline  test 


schedules,  thereby  increasing  the  acquisition  timeline 
and  program  test  costs. 

DoD  policy  should  mandate  integrated  test  planning 
and  execution  on  all  programs  to  the  extent  possible. 
To  accomplish  this,  programs  must  establish  a  team 
made  up  of  all  relevant  organizations  (including 
contractors,  developmental  and  operational  test  and 
evaluation  communities)  to  create  and  manage  the 
approach  to  incorporate  integrated  testing  into  the 
T8cE  Strategy  and  the  TEMP. 

Findings: 

•  Service  acquisition  programs  are  incorporating 
integrated  testing  to  a  limited  degree  through 
varying  approaches. 

•  Additional  emphasis  on  integrated  testing  will 
result  in  greater  T&E  process  efficiency  and 
program  cost  reductions. 

Recommendations: 

•  Implement  OSD  and  Service  policy  mandating 
integrated  DT&E/OT&E  planning  and  execu¬ 
tion  throughout  the  program: 

-  Require  sharing  and  access  to  all  appropriate 
system-level  and  selected  component-level 
test  and  model  data  by  government  DT  and 
OT  organizations,  as  well  as  the  prime 
contractor,  where  appropriate; 

-  Integrate  test  events,  where  practical,  to 
satisfy  OT  and  DT  requirements. 

Operational  test  readiness  review  (OTRR) 

Each  Service  has  an  operational  test  readiness  review 
(OTRR)  process.  Although  it  varies  from  Service  to 
Service,  the  process  generally  results  in  in-depth 
reviews  of  readiness  to  undergo  an  lOT&E  event. 

Findings: 

•  Shortcomings  in  system  performance,  suitability, 
and  RAM  are  usually  identified  during  the  OTRR. 

•  In  most  cases,  the  operational  test  readiness 
certifying  authority  is  well  aware  of  the  risk  of  not 
meeting  OT  criteria  when  major  shortcomings  exist. 

•  Because  of  funding  constraints,  the  low  priority 
given  to  sustainment,  as  well  as  the  urgency  in 
recent  years  to  get  new  capabilities  to  the 
warfighter,  major  suitability  shortcomings  have 
rarely  delayed  the  commencement  of  dedicated 

lOT&E. 

Recommendations: 

•  Conduct  periodic  operational  assessments  to 
evaluate  progress  and  the  potential  for  achieving 
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predetermined  entrance  criteria  for  operational 
test  events. 

•  Conduct  an  independent  assessment  of  opera¬ 
tional  test  readiness  (AOTR)  prior  to  the  OTRR. 

•  Include  a  detailed  RAM  template  in  preparation 
for  the  OTRR. 

•  Require  the  Command  Acquisition  Executive 
(CAE)  to  submit  a  report  to  OSD  that  provides 
the  rationale  for  the  readiness  decision. 

OSD  test  and  evaluation  organization 

The  Task  Force  was  asked  to  assess  OSD  roles  and 
responsihilities  for  T&E  oversight.  TScE  has  been  a 
visible  part  of  OSD  since  the  early  1970s,  reporting  to 
the  Research  and  Engineering  command  section  when 
it  was  in  charge  of  acquisition  oversight  and  subse¬ 
quently  to  the  Under  Secretary  of  Defense  for 
Acquisition  (now  ATScL).  The  early  TScE  office  was 
responsible  for  all  T&E,  ranges,  resources  oversight,  and 
policy.  In  1983,  Congress  established  an  independent 
Director,  Operational  Test  and  Evaluation  (DOT&E) 
organization,  reporting  directly  to  the  Secretary  of 
Defense  (SECDEF),  responsible  for  operational  test 
and  evaluation  policy,  budget  review,  and  assessments  of 
operational  effectiveness  and  suitability.  The  Live  Fire 
Test  (LET)  oversight  function  was  created  and  added  to 
the  DT&,E  office  responsibilities  in  the  mid-1980s. 
Later,  the  LET  oversight  function  was  moved  to  the 
DOT&,E  organization. 

In  1999,  the  DT&E  organization  was  dismantled  by 
DoD.  Many  functions  were  moved  to  DOT&,E, 
including  test  ranges  and  resources,  and  joint  T&E 
oversight.  Some  of  the  remaining  T&E  personnel  billets 
were  eliminated  to  comply  with  a  congressionally 
mandated  (AT&L)  acquisition  staff  reduction.  The 
residual  DT&E  pohcy  and  oversight  functions  were 
separated  and  moved  lower  in  the  AT&L  organization. 

A  2000  DSB  Task  Force  Study  on  Test  and 
Evaluation  Capabilities  recommended  that  DoD  create 
a  test  and  evaluation  resource  enterprise  within  the 
office  of  the  DOT&E  to  provide  more  centralized 
management  of  T&E  facilities.  This  recommendation 
ultimately  led  to  removing  the  test  ranges  and 
resources  oversight  from  DOT&E,  abandoning  the 
notion  of  centralized  management,  and  the  establish¬ 
ment  of  the  Test  Resource  Management  Center 
(TRMC)  in  AT&L  (as  directed  by  the  National 
Defense  Authorization  Act  for  Fiscal  Year  2003). 

Findings: 

Current  policy  as  of  December  2007  mandates  that 
developmental  and  operational  test  activities  be 
integrated  and  seamless  throughout  the  system  life 


cycle.  There  must  be  enough  experts  in  OSD  with  the 
ability  to  understand  and  articulate  lessons  learned  in 
early  testing  and  the  ability  to  execute  the  new  T&E 
policy.  That  policy  is  to  “take  into  account  all  available 
and  relevant  data  and  information  from  contractors 
and  government  sources”  in  order  to  “maximize  the 
efficiency  of  the  T&E  process  and  effectively  integrate 
developmental  and  operational  T&E.” 

•  Currently  there  is  not  an  OSD  organization  with 
comprehensive  DT  oversight  responsibility,  au¬ 
thority,  or  staff  to  coordinate  with  the  operational 
test  office: 

-  The  historic  DT  organization  has  been 
broken  up  and  residual  DT  functions  were 
moved  lower  in  the  organization  in  1999,  and 
lower  yet  in  2002; 

-  Programmatic  DT  oversight  is  limited  by  staff 
size  and  often  performed  by  generalists  vice 
T&E  experts; 

-  Recruitment  of  senior  field  test  personnel  is 
hampered  by  DT’s  organizational  status; 

-  Existing  residual  organizations  are  fragment¬ 
ed  and  lack  clout  to  provide  DT  guidance; 

-  System  performance  information  and  DT 
lessons  learned  across  DoD  have  been  lost; 

-  DT  is  not  viewed  as  a  key  element  in  AT&L 
system  acquisition  oversight; 

-  Documentation  of  DT  results  by  OSD  is 
minimal. 

•  Access  to  models,  data,  and  analysis  results  is 
restricted  by  current  practice  in  acquisition 
contracting,  and  by  the  lack  of  expertise  in  the 
DT  organization. 

•  TRMC  has  minimal  input  to  program- specific 
questions  or  interaction  with  oversight  organiza¬ 
tions  on  specific  programs. 

-  Organizational  separation  is  an  impediment. 

Recommendations: 

•  Implementation  of  integrated  and  seamless  DT 
and  OT  will  require,  at  a  minimum,  greater 
coordination  and  cooperation  between  all  testing 
organizations. 

•  Consolidate  DT-related  functions  in  AT&L  to 
help  reestablish  a  focused,  integrated,  and  robust 
organization: 

-  Reestablish  program  oversight  and  policy,  and 
Foreign  Comparative  Test  (FCT); 

-  Have  Director,  DT&E  directly  report  to 
Deputy  Under  Secretary  of  Defense,  Acqui¬ 
sition  and  Technology  (DUSD[A&T]); 

-  Restore  TEMP  approval  authority  to  Direc¬ 
tor,  DT&E. 
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•  Integrate  TRMC  activities  early  into  DT  pro¬ 
gram  planning: 

-  Make  TRMC  responsible  for  reviewing  the 
resources  portion  of  the  TEMP. 

•  If  such  an  organization  is  established  and  proves 
itself  effective,  consider  as  part  of  a  future 
consolidation  moving  LET  back  to  its  original 
DT  location. 

The  LET  change  requires  the  concurrence  of 
DOT&E  and  a  legislative  change  to  Title  10  because 
of  the  change  in  reporting  official.  All  the  other 
recommendations  made  throughout  the  report  can  be 
implemented  within  current  DoD  authority. 

Other  issues 

Several  other  issues  were  addressed  as  a  part  of  the 
study.  A  discussion  of  each  of  the  following  topics, 
along  with  findings  and  recommendations,  may  be 
found  in  the  body  of  the  report. 

•  Program  Structure 

•  Requirements  Definition 

•  Contractual  Performance  Requirements 

•  Alignment  of  DoD  Technology  with  Systems 
Engineering  Procedures 

•  Commercial  Off-The-Shelf 

•  Systems  of  Systems 

Summary  and  implementation  status 

In  summary,  the  single  most  important  step  required 
to  remedy  the  high  suitability  failure  rates  is  to  insure 
that  programs  are  formulated  to  execute  a  viable 
systems  engineering  strategy  from  the  beginning, 
including  a  robust  RAM  program,  as  an  integral  part 
of  design  and  development.  A  second  and  related 
priority  is  to  ensure  that  government  organizations 
reconstitute  a  cadre  of  experienced  T&E,  engineering 
and  RAM  personnel  to  support  the  acquisition  process. 
A  third  priority  is  to  integrate  developmental  and 
operational  testing  to  the  extent  practicable.  A 
Reliability  Improvement  Working  Group  was  estab¬ 
lished  in  March  2008  to  address  these  three  issues. 

The  reliability  subgroup  worked  on  developing  a 
reliability  acquisition  policy  and  framework  that 
includes  consistent,  concise  sample  REP  language  that 
will  encourage  developers  to  plan  for  and  resource  a 
reliability  grovrth  program  as  a  part  of  design  and 
development;  Phased  templates  to  evaluate  RAM 
activities  throughout  program  reviews;  and  Standard 
evaluation  criteria  to  provide  a  consistent  way  to 
evaluate  an  acquisition  program’s  reliability  health 
throughout  the  development  process.  On  July  21,  2008 
the  Undersecretary  of  Defense  for  Acquisition,  Tech- 
nology,  and  Logistics  signed  a  policy  memo  on 


Reliability,  Availability,  and  Maintainability  which 
implements  the  key  RAM  recommendations  in  the 
DSB  report. 

The  personnel  subgroup  addressed  four  major  issues: 
first,  a  policy  to  enable  workforce  reconstitution;  second, 
a  plan  to  reconstitute  RAM  and  T&E  personnel  where 
necessary,  third,  training  and  education  for  RAM  and 
T&E  personnel;  and  fourth,  establishing  and  staffing 
Centers  of  Excellence  and  expertise. 

The  integrated  testing  subgroup  developed  guidehnes 
for  early  involvement  in  requirements  and  REP 
development;  contractual  language  for  data  access  and 
sharing;  and  synchronization  of  the  Test  and  Evaluation 
Master  Plan  (TEMP)  and  Systems  Engineering  Guide. 

Eurther  implementation  of  these  and  other  recom¬ 
mendations  in  the  report  will  not  be  easy,  but  will  pay 
large  dividends  in  improvements  to  the  acquisition 
process  and  reduced  life  cycle  costs.  □ 
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